Fast Fault Injection with Virtual Machines

Fault injection is a widely used technique to test the robustness of software systems and to find bugs in applications. We focus on using fault injection to judge the quality of error handling in applications. Even mature applications have serious bugs in error handling [3]. To find all bugs related to bad error handling when using a certain API one has to do exhaustive fault injection, i.e., to inject all error values found in the API’s specification. When doing exhaustive fault injection one has to do one fault injection experiment per API usage of the application to test. That means one need to run the application once per fault injection. So the run-time of all experiments together grows like O(n) in terms of API usages. One can reduce run-time complexity to O(n) by utilizing snapshots and rollbacks [2, 4]. The idea is to make a snapshot of the application directly before a fault is injected. After the fault injection experiment is finished the application’s state is rolled back to the last snapshot. In this way one can reuse the progress until the fault injection an application has already achieved. Figure 1 illustrates such a single fault injection run. The original fault free execution of the application is bold. The run has 3 API usages. Each API usage is a fault injection point. Due to an API usage, e.g., a function call, can fail for multiple reasons usually more than one fault needs to be injected per fault injection point to do an exhaustive analysis. In the example, at the first and the third fault injection point two faults need to be injected and at the second one three faults. The different faults are derived from the specified error values of the called API functions. In our experiments we are not interested in a complete run after the fault has been injected. It is sufficient to execute a limited number of instructions after injecting the fault, e.g., until the currently executed function returns to its caller. That is why, the runtime complexity is O(n) instead of O(n) when doing each experiment in a separate run. To underpin the complexity issues, we have parsed Figure 1. A fault injection run with 3 fault injection points with 2, 3, and 2 different error values to injection.